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Air  Force  Level  I  Logistics  Year  2000 
End-to-End  Test  Planning 


Executive  Summary 


Introduction.  This  is  one  in  a  series  of  reports  being  issued  by  the  Inspector  General, 
DoD,  in  accordance  with  an  informal  partnership  with  the  Chief  Information  Officer, 
DoD,  to  monitor  DoD  efforts  to  address  the  year  2000  computing  challenge.  For  a 
complete  listing  of  audit  projects  addressing  the  issue,  see  the  year  2000  web  pages  on 
the  IGnet  at  http :  // w w w .  ignet .  gov . 

The  DoD  Year  2000  Management  Plan  (DoD  Management  Plan)  assigns  responsibility 
to  the  Principal  Staff  Assistants  for  ensuring  the  end-to-end  functional  process  flows 
that  support  their  functional  area  are  assessed  either  in  a  Joint  Staff  or  commander  in 
chief  year  2000  operational  evaluation,  a  Service-sponsored  system  integration  test,  or  a 
functional  area  year  2000  end-to-end  test.  The  Principal  Staff  Assistants  are  also 
responsible  for  planning,  executing,  and  evaluating  all  mission-critical  systems  not 
otherwise  tested  and  ensuring  that  processes  that  fall  within  their  purview  are  evaluated. 
The  Deputy  Under  Secretary  of  Defense  (Logistics  and  Materiel  Readiness) 
(DUSD[L&MR])  acts  on  behalf  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  the  Principal  Staff  Assistant  for  logistics,  in  performing 
those  functions  for  the  logistics  functional  area.  Logistics  end-to-end  test  planning  was 
accomplished  through  the  “Logistics  Capstone  Operational  Assessment  Plan  for  Year 
2000”  (Logistics  Capstone  Plan). 

Logistics  functional  end-to-end  testing  was  divided  into  three  phases.  Level  I  was  intra- 
Component  testing,  and  Level  II  was  inter-Component  testing.  Level  III  testing  was  to 
be  conducted  as  required  to  perform  retesting.  The  DUSD(L&MR)  provided  oversight 
for  Level  II  testing  while  delegating  responsibility  for  execution  of  Level  I  testing  to  the 
Components.  Air  Force  Level  I  testing  began  in  February  1999  and  was  completed  in 
late  October  1999. 

Objective.  The  audit  objective  was  to  evaluate  the  effectiveness  of  the  year  2000  end- 
to-end  tests  planned  for  the  logistics  functional  area.  This  report,  the  sixth  in  a  series 
on  logistics  end-to-end  testing,  addresses  the  overall  Level  I  end-to-end  test  planning 
accomplished  by  the  Air  Force.  Level  II  end-to-end  testing  for  the  Air  Force  was 
addressed  in  Inspector  General,  DoD,  Report  No.  00-021,  “Air  Force  Logistics 
Year  2000  End-to-End  Test  Planning,”  October  26,  1999. 


Results.  The  Air  Force  Level  I  end-to-end  test  planning  for  core  logistics  processes 
did  not  meet  the  requirements  outlined  in  the  DoD  Management  Plan  and  the  Logistics 
Capstone  Plan.  Although  the  Air  Force  identified  22  core  logistics  processes  that  were 
critical  to  the  Air  Force  and  planned  to  test  14  of  the  core  processes  and  22  mission- 
critical  logistics  systems  during  Level  I  end-to-end  testing,  the  Air  Force  did  not 
'  develop  a  sufficiently  detailed  plan  for  conducting  the  tests.  As  a  result,  the  ability  to 
analyze  the  sufficiency  of  required  testing  of  core  logistics  processes  and  mission- 
critical  logistics  systems  was  hampered.  As  of  October  27,  1999,  the  Air  Force  had 
prepared  test  results  for  only  3  of  7  planned  test  scenarios  involving  7  of  the  14  core 
processes  and  21  of  the  22  mission-critical  systems.  However,  the  Air  Force  did  plan 
to  perform  the  verification  and  validation  of  100  percent  of  mission-critical  code.  See 
the  Finding  section  for  details. 

Summary  of  Recommendations.  We  recommend  that  the  Chief  Information  Officer, 
Department  of  the  Air  Force,  ensure  that  contingency  plans  for  the  22  mission-critical 
logistics  systems  that  were  to  be  included  in  Level  I  end-to-end  testing  are  tested.  We 
also  recommend  that  the  Chief  Information  Officer  ensure  that  a  risk  management  plan 
that  includes  a  risk  assessment  and  mitigation  plan  for  each  of  the  Air  Force  core 
logistics  processes  is  developed. 

Management  Comments.  A  draft  of  this  report  was  issued  on  November  5,  1999. 
The  Air  Force  did  not  respond  to  the  draft  report.  We  request  that  the  Chief 
Information  Officer,  Department  of  the  Air  Force,  provide  written  comments  on  this 
final  report  by  December  15,  1999. 
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Background 


Executive  Order.  Because  of  the  potential  failure  of  computers  to  function 
throughout  the  Government,  the  President  issued  Executive  Order  13073,  “Year 
2000  Conversion,”  February  4,  1998,  making  it  policy  that  Federal  agencies 
ensure  that  no  critical  Federal  program  experiences  disruption  because  of  the 
year  2000  (Y2K)  problem.  The  order  requires  that  the  head  of  each  agency 
ensure  that  efforts  to  address  the  Y2K  problem  receive  the  highest  priority 
attention  in  the  agency. 

DoD  Y2K  Management  Strategy.  In  his  role  as  the  DoD  Chief  Information 
Officer,  the  Senior  Civilian  Official,  Office  of  the  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence),  issued  the 
“DoD  Year  2000  Management  Plan,  Version  2.0”  (DoD  Management  Plan)  in 
December  1998.  The  DoD  Management  Plan  required  DoD  Components  to 
implement  a  five-phase  (awareness,  assessment,  renovation,  validation,  and 
implementation)  Y2K  management  process  to  be  completed  by  December  31, 
1998,  for  mission-critical  systems. 

The  DoD  Management  Plan  also  provides  guidance  for  implementing  the 
Deputy  Secretary  of  Defense  memorandum,  “Year  2000  (Y2K)  Verification  of 
National  Security  Capabilities,”  August  24,  1998,  that  requires  that  each 
Principal  Staff  Assistant  (PSA)  of  the  Office  of  the  Secretary  of  Defense  “verify 
that  all  functions  under  his  or  her  purview  will  continue  unaffected  by  Y2K 
issues.”  That  verification  was  to  be  performed  after  completion  of  the 
five-phase  management  approach  that  culminated  with  completion  of  the 
implementation  phase,  December  31,  1998.  That  further  testing,  to  be 
conducted  during  the  first  half  of  1999,  was  planned  and  conducted  from  a 
mission  perspective  rather  than  a  system  perspective  and  would  increase  the 
confidence  that  any  errors  or  omissions  in  system  remediation  would  be  found. 
The  Deputy  Under  Secretary  of  Defense  (Logistics  and  Materiel  Readiness) 
(DUSD[L&MR])  acts  on  behalf  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  the  PSA  for  logistics. 

DoD  Logistics  End-to-End  Planning.  The  DUSD(L&MR)  implemented  and 
executed  key  components  of  the  DoD  Management  Plan  in  his  efforts  to 
adequately  plan  for  and  manage  logistics  functional  end-to-end  testing.  Test 
planning  was  accomplished  through  the  “Logistics  Capstone  Operational 
Assessment  Plan  for  Year  2000”  (Logistics  Capstone  Plan),  dated  October  30, 
1998,  and  approved  in  November  1998.  The  Logistics  Capstone  Plan  provided 
the  overall  strategy  for  conduct  of  the  logistics  end-to-end  testing  and  was 
coordinated  with  the  Services,  the  Defense  Logistics  Agency,  the  Joint 
Interoperability  Test  Command,  and  the  Joint  Staff.  The  October  1998 
Logistics  Capstone  Plan  was  updated  in  February  1999  and  again  in  May  1999 
to  reflect  evolving  schedules  and  processes.  Its  name  was  changed  to  “Logistics 
Capstone  Plan  for  Year  2000  End-to-End  Test”  as  part  of  the  February  update. 
In  this  report,  unless  otherwise  noted,  Logistics  Capstone  Plan  refers  to  the 
May  20,  1999,  version. 
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Objective 


The  audit  objective  was  to  evaluate  the  effectiveness  of  the  Y2K  end-to-end  tests 
planned  for  the  logistics  functional  area.  This  report,  the  sixth  in  a  series  on 
logistics  end-to-end  testing,  addresses  the  overall  Level  I  end-to-end  test 
planning  accomplished  by  the  Air  Force.  Level  II  end-to-end  testing  for  the  Air 
Force  was  addressed  in  Inspector  General,  DoD,  Report  No.  00-021,  “Air 
Force  Logistics  Year  2000  End-to-End  Test  Planning,”  October  26,  1999.  See 
Appendix  A  for  a  discussion  of  the  audit  scope  and  methodology  and 
Appendix  B  for  a  summary  of  prior  coverage. 


Air  Force  Planning  for  Level  I  Logistics 
Functional  End-to-End  Testing 

The  Air  Force  Level  I  end-to-end  test  planning  for  core  logistics 
processes  did  not  meet  the  requirements  outlined  in  the  DoD 
Management  Plan  and  the  Logistics  Capstone  Plan.  Although  the  Air 
Force  identified  22  core  processes  that  were  critical  to  the  Air  Force  and 
planned  to  test  14  of  the  core  processes  and  22  mission-critical  logistics 
systems  during  Level  I  testing,  the  Air  Force  did  not  develop  sufficiently 
detailed  plans  for  conducting  the  tests.  As  a  result,  the  ability  to  analyze 
the  sufficiency  of  required  testing  of  core  processes  and  mission-critical 
systems  was  hampered.  As  of  October  27,  1999,  the  Air  Force  had 
completed  test  results  for  only  3  of  7  planned  test  scenarios  involving 
7  of  the  14  core  processes  and  21  of  the  22  mission-critical  systems. 
However,  the  Air  Force  did  plan  to  perform  the  verification  and 
validation  of  100  percent  of  mission-critical  code. 


End-to-End  Test  Guidance 

The  Logistics  Capstone  Plan  provides  the  overall  strategy  for  conduct  of  the 
DoD  logistics  end-to-end  testing.  The  Logistics  Capstone  Plan  defines  three 
levels  of  testing  and  delegates  responsibility  for  each.  The  multilevel  test 
approach  consisted  of  intra-Component  events  (Level  I),  inter-Component 
events  (Level  II),  and  post-test  activities  that  include  retest  (Level  III).  Level  I 
tests  were  designed  to  ensure  processes  and  systems  within  a  Component’s 
organizational  boundaries  are  Y2K  ready.  Level  II  testing  was  to  verily  core 
processes  and  information  flows  that  involve  more  than  a  single  Component  are 
Y2K  ready.  The  execution  and  oversight  of  the  Level  I  testing  was  delegated  to 
the  Components  while  DUSD(L&MR)  focused  on  the  Level  II  testing  and  post¬ 
test  events,  such  as  retest,  during  Level  III. 


Air  Force  Planning  for  End-to-End  Testing 

The  Air  Force  Level  I  end-to-end  test  planning  for  core  logistics  processes  did 
not  meet  the  requirements  outlined  in  the  DoD  Management  Plan  and  the 
Logistics  Capstone  Plan.  The  Air  Force  did  not  issue  a  test  plan  specifically  for 
Level  I  end-to-end  testing.  Therefore,  our  review  was  limited  to  an  evaluation 
of  the  “U.S.  Air  Force  Logistics  Year  2000  Level  II  End-to-End  Test  Plan”  (the 
Air  Force  Level  II  Test  Plan),  version  3.6.3,  June  8,  1999,  and  the  draft  “Air 
Force  Year  2000  Assessment  Plan  for  Integrated  Logistics”  (the  Air  Force 
Assessment  Plan),  version  1.2,  February  17,  1999.  The  Air  Force  implemented 
the  Logistics  Capstone  Plan  regarding  Level  II  end-to-end  testing  with  the 
issuance  of  the  Air  Force  Level  II  Test  Plan.  However,  the  Air  Force  Level  II 
Test  Plan,  which  defines  the  Air  Force  strategy  for  its  participation  in  the 
logistics  end-to-end  testing,  does  not  fully  address  Level  I  logistics  end-to-end 
testing. 
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The  Air  Force  Level  II  Test  Plan  and  the  draft  Air  Force  Assessment  Plan  were 
prepared  in  response  to  the  DUSD(L&MR)  tasking  to  support  inter-Service 
assessment  of  standard  logistics  information  systems.  The  Logistics  Capstone 
Plan  addresses  the  development  of  detailed  plans  for  assessing  mission-critical 
systems.  The  Air  Force  Level  II  Test  Plan  and  the  draft  Air  Force  Assessment 
Plan  did  not  fully  address  that  guidance  concerning  the  development  of  plans. 

•  They  address  assessment  requirements  for  logistics  processes  that  are 
within  the  Air  Force  integrated  logistics  area  of  responsibility. 

•  They  provide  only  limited  detail  regarding  the  strategy  for 
conducting  logistics  end-to-end  testing. 

•  They  do  not  clearly  define  roles  and  responsibilities  for  conducting 
Level  I  end-to-end  tests;  provide  specific  time  frames  for  completing 
the  tests;  and  provide  details  regarding  data  collection  and  analyses, 
which  the  Logistics  Capstone  requires. 

Roles  and  Responsibilities.  The  Air  Force  Level  II  Test  Plan  and  the  draft  Air 
Force  Assessment  Plan  did  not  clearly  define  roles  and  responsibilities  for 
conducting  Level  I  end-to-end  testing.  The  Air  Force  Level  II  Test  Plan  stated 
that  the  Air  Force  Central  Design  Activities  at  the  Materiel  Systems  Group, 
Wright-Patterson  Air  Force  Base,  Ohio,  and  the  Standard  Systems  Group, 
Maxwell  Air  Force  Base-Gunter  Annex,  Alabama  (the  Central  Design 
Activities),  were  responsible  for  conducting  Level  I  testing  of  core  processes 
and  mission-critical  systems.  However,  there  was  no  detailed  plan  on  the  Air 
Force  activities,  mission-critical  systems,  or  information  flows  that  would  be 
tested.  The  draft  Air  Force  Assessment  Plan  provided  only  general  guidance, 
stating  that  the  installations  and  logistics  systems  program  office  integrated 
process  team  with  members  from  all  Air  Force  installations  and  logistics 
directorates,  in  conjunction  with  the  Y2K  offices  of  the  domains  being  tested, 
will  assign  responsibilities  to  the  appropriate  organization  within  the  domain 
being  tested. 

Time  Frames.  The  Air  Force  did  not  establish  a  specific  time  frame  for 
performing  Level  I  end-to-end  testing.  The  Air  Force  Level  II  Test  Plan  and 
the  draft  Air  Force  Assessment  Plan  did  not  state  when  Level  I  end-to-end 
testing  was  to  begin.  They  simply  stated  that  Level  I  end-to-end  testing  would 
end  on  May  5,  1999.  According  to  Air  Force  officials,  Level  I  end-to-end 
testing  of  its  core  processes  and  mission-critical  systems  began  in  February  1999 
and  was  actually  completed  in  late  October  1999. 

Data  Collection  and  Analyses.  The  Air  Force  Level  II  Test  Plan  and  the  draft 
Air  Force  Assessment  Plan  did  not  specifically  state  how  data  regarding  Level  I 
end-to-end  testing  would  be  collected  and  analyzed.  They  included  no  detailed 
data  collection  procedures  to  provide  instructions  to  test  teams  regarding  data 
collection,  processing,  and  analysis.  Also,  they  included  no  detail  on  how  data 
collected  would  be  summarized  for  reporting  to  higher  level  Air  Force  officials. 

The  draft  Air  Force  Assessment  Plan  stated  that  it  was  the  responsibility  of  the 
installations  and  logistics  systems  program  office  integrated  process  team,  along 
with  the  Y2K  offices  of  the  domains  being  tested,  to  aggregate,  analyze,  and 
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report  the  results  of  the  tests.  The  same  groups  were  responsible  for  briefing 
the  leadership  of  the  Air  Force  at  appropriate  intervals  that  were  left  undefined. 
The  Logistics  Capstone  Plan  required  that  a  data  collection  and  analysis  strategy 
be  developed  that  provided  for  sufficient  information  to  support  the  design  of 
the  end-to-end  tests,  capture  test  results,  and  support  post-testing  activities. 
Post-testing  activities  include  final  analysis  of  test  results,  tracking  the 
implementation  of  corrective  actions  for  failures  discovered  during  tests,  and 
providing  supporting  back-up  data  for  corporate-level  assessment  reports.  The 
type  of  data  and  how  it  will  be  collected  should  be  documented  in  a  data 
collection  and  analysis  plan.  Without  data  collection  and  analysis  plans,  there 
was  no  organized  or  standardized  approach  among  the  participating  systems, 
and  there  was  no  assurance  that  test  goals  would  be  met  and  that  testing  would 
be  successfully  accomplished. 


Testing  of  Processes  and  Systems 

The  Air  Force  planned  to  test  14  of  22  core  logistics  processes  and  22  mission- 
critical  logistics  systems  during  Level  I  end-to-end  testing.  The  general 
approach  taken  by  the  Air  Force  was  to  identify  critical  functional  processes  and 
then  identify  the  systems  that  supported  those  processes. 

Core  Processes  Tested.  The  draft  Air  Force  Assessment  Plan  identified  22 
core  logistics  processes  that  were  critical  to  the  Air  Force.  Twelve  core 
processes  were  determined  to  be  the  most  critical.  The  initial  plan  was  to  test 
the  12  most  critical  processes  during  Level  I  and  Level  II  end-to-end  testing. 
Testing  of  the  remaining  10  core  processes  would  be  based  on  the  results  of  the 
initial  test  of  the  12  core  processes;  the  results  of  applying  Y2K  tools,  such  as 
code  screening,  to  the  mission-critical  systems;  the  results  of  multiple  functional 
testing  in  the  test  facility;  the  results  of  the  independent  verification  and 
validation  process  review;  and  the  availability  of  the  crisis  action  team. 

Although  the  Air  Force  initially  identified  12  core  processes  for  testing,  14  core 
processes  were  included  in  Level  I  end-to-end  testing.  However,  the  Air  Force 
did  not  document  its  rationale  for  prioritizing  and  selecting  the  14  core 
processes  for  evaluation  as  required  by  the  Logistics  Capstone  Plan.  The 
Central  Design  Activities  consolidated  the  14  core  processes  into  7  test 
scenarios.  The  Materiel  Systems  Group  and  Standard  Systems  Group  were 
responsible  for  testing  the  core  processes.  Appendix  C  contains  a  list  of  the 
22  core  processes  and  the  7  test  scenarios  and  14  core  processes  actually 
selected  by  the  Air  Force  for  Level  I  end-to-end  testing.  Appendix  D  contains  a 
list  of  the  22  mission-critical  logistics  systems  and  the  7  test  scenarios  that  were 
to  be  included  in  Level  I  end-to-end  testing. 

Systems  Tested.  Of  the  37  mission-critical  logistics  systems  listed  in  the  DoD 
Y2K  Reporting  Database  as  of  October  1,  1999,  the  Air  Force  identified 
22  mission-critical  systems  that  supported  the  14  core  processes  and  planned  to 
test  them  during  Level  I  end-to-end  testing.  Of  the  22  mission-critical  systems, 
4  were  also  evaluated  during  Level  II  end-to-end  testing;  14  were  also  in,  or 
planned  to  be  in,  operational  evaluations  or  a  Service-sponsored  systems 
integration  test. 
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Testing  of  39  mission-critical  logistics  systems  was  addressed  in  Inspector 
General,  DoD,  Report  No.  00-021,  “Air  Force  Logistics  Year  2000  End-to-End 
Test  Planning,”  October  26,  1999.  Prior  to  the  issuance  of  that  final  report,  the 
Air  Force  reclassified  the  Embedded  Global  Positioning  System/Inertial 
Navigation  System  and  the  Improved  Maintenance  Management  Program,  and 
they  are  no  longer  included  in  the  DoD  Y2K  Reporting  Database  as  mission- 
critical  logistics  systems.  The  Embedded  Global  Positioning  System/Inertial 
Navigation  System  was  changed  from  a  logistics  to  a  weapon  system.  The 
Improved  Maintenance  Management  Program  was  determined  not  to  be  a 
mission-critical  system. 

Test  Results.  The  Air  Force  had  not  developed  a  detailed  Level  I  end-to-end 
test  plan  that  articulated  test  objectives  and  processing  of  results  as  outlined  in 
the  DoD  Management  Plan  and  the  Logistics  Capstone  Plan.  Also,  reports  for 
all  completed  tests  had  not  been  prepared  as  required.  Therefore,  we  could  not 
determine  the  status  of  the  Level  I  end-to-end  testing  or  whether  the  objectives 
of  the  test  had  been  met. 

As  of  late  October  1999,  results  for  only  three  of  seven  test  scenarios  had  been 
provided,  even  though  the  test  schedule  provided  by  the  Air  Force  showed  that 
all  seven  test  scenarios  were  scheduled  to  be  completed  by  August  20,  1999. 

The  DoD  Management  Plan  calls  for  final  test  reports  to  be  completed  within 
30  days  of  completion  of  testing.  Air  Force  personnel  at  the  system  developing 
commands,  the  Materiel  Systems  Group  and  the  Standard  Systems  Group, 
performed  the  independent  verification  and  validation  for  the  Air  Force  Level  I 
end-to-end  tests  that  were  completed.  However,  because  personnel  responsible 
for  the  independent  verification  and  validation  of  end-to-end  tests  are  also 
integral  parts  of  the  commands  performing  the  tests,  this  could  be  perceived  as  a 
lack  of  overall  independence  in  the  verification  and  validation  of  the  end-to-end 
tests.  The  Logistics  Capstone  Plan  states  that  Components  must  use  independent 
agents  to  verify  the  test  results. 

Results  from  the  3  test  scenarios  showed  that  21  of  the  22  mission-critical 
systems  scheduled  for  Level  I  testing  were  tested.  Test  Scenario  No.  1  included 
two  core  processes,  aerospace  equipment  maintenance  process  and 
engine/module  requisition  process,  and  eight  mission-critical  systems.  Test 
Scenario  No.  2  included  four  core  processes,  inventory  management  process, 
new  workload  process,  requisition  management  process,  and  wartime  processing 
scenario,  and  11  mission-critical  systems.  Test  scenario  No.  4  included  the 
deployment  management  process  and  three  mission-critical  systems.  The  Core 
Automated  Maintenance  System  was  included  in  Test  Scenario  No.  1  and  Test 
Scenario  No.  2.  Results  for  the  three  test  scenarios  indicated  no  Y2K-related 
anomalies. 


Contingency  Plans  and  Risk  Management 

In  addition  to  not  having  a  Level  I  end-to-end  test  plan  that  clearly  addressed 
roles  and  responsibilities,  provided  specific  time  frames  for  completing  the 
testing,  and  provided  details  regarding  data  collection  and  analysis,  the  Air 
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Force  Level  II  Test  Plan  and  the  draft  Air  Force  Assessment  Plan  did  not  . 
address  testing  of  contingency  plans  or  measures  to  minimize  risk  of  Y2K- 
related  system  failures. 

Contingency  Plans.  The  DoD  Management  Plan  stated  that  Y2K  system 
contingency  plans  are  required  for  all  mission-critical  systems  and  that  all 
contingency  plans  should  be  validated  or  exercised  to  ensure  potential  actions 
are  executable.  The  DoD  Management  Plan  established  a  target  date  of 
June  30,  1999,  for  exercising  of  both  system  and  operational  contingency  plans. 
However,  the  Logistics  Capstone  Plan  extended  the  target  completion  date  for 
exercising  of  individual  contingency  plans  to  September  1,  1999. 

In  a  September  27,  1999,  reply  to  a  draft  of  Inspector  General,  DoD,  Report 
No.  00-021,  the  Air  Force  stated  that  it  was  testing  contingency  plans  for 
mission-critical  logistics  systems  as  part  of  Level  I  testing.  The  Air  Force 
further  stated  that  it  may  test  contingency  plans  for  other  mission-critical 
systems,  depending  on  the  coverage  and  results  that  are  achieved  during  the 
Level  I  tests.  However,  the  Air  Force  did  not  estimate  the  completion  date  for 
exercising  contingency  plans  during  Level  I  tests.  Discussions  with  Air  Force 
officials  did  not  clarify  the  Air  Force  intentions  on  exercising  the  contingency 
plans  for  mission-critical  systems  involved  or  not  involved  in  Level  I  testing. 

To  reduce  the  risk  that  Y2K-related  failures  will  impair  mission  capabilities,  the 
Air  Force  needs  to  test  contingency  plans  for  all  22  mission-critical  systems  to 
ensure  that  adequate  workarounds  are  in  place  in  the  event  of  Y2K-related 
system  failures.  The  DoD  Management  Plan  states  that  Y2K  functional  and 
operational  end-to-end  test  exercises  will  be  used  to  evaluate  the  Y2K 
contingency  plans  of  designated  mission-critical  systems  and  will  contribute  to  a 
complete  evaluation  of  DoD  operational  capability. 

Risk  Assessments.  The  Air  Force  did  not  document  the  risk  assessments 
performed  during  the  process  of  prioritizing  logistics  processes  for  inclusion  in 
end-to-end  testing  as  required  by  the  DoD  Management  Plan.  The  DoD 
Management  Plan  states  that  the  Y2K  event  master  planning  sessions  were  to 
identify  and  prioritize  core  processes  and  perform  risk  assessments.  The 
Logistics  Capstone  Plan  identified  four  general  categories  of  corporate-level 
risk:  funding;  scheduling;  scope  of  testing;  and  test  environment.  It  also 
assigned  each  category  a  risk  rating  of  high,  medium,  or  low,  based  on 
probability  of  occurrence  and  consequences  of  occurrence,  as  well  as  listed  the 
mitigation  of  a  particular  risk.  The  Logistics  Capstone  Plan  states  that  the 
discussion  of  corporate-level  risks  is  an  initial  risk  assessment.  In  addition,  the 
Logistics  Capstone  Plan  states  that  a  complete  risk  mitigation  plan  will  be 
incorporated  in  an  overall  risk  management  plan.  DUSD(L&MR)  had  planned 
to  draft  a  risk  management  plan  on  all  core  logistics  processes  by  September 
1999.  The  draft  Air  Force  Assessment  Plan  did  not  include  guidance  on 
preparing  or  submitting  a  risk  management  plan  to  DUSD(L&MR)  for  the  Air 
Force  core  logistics  processes  and  systems.  As  a  result,  as  of  October  22, 

1999,  the  Air  Force  had  not  completed  a  risk  management  plan  for  review  and 
inclusion  in  the  overall  DUSD(L&MR)  risk  management  plan.  Therefore, 
DUSD(L&MR)  did  not  have  sufficient  information  to  complete  a  risk 
management  plan  for  all  core  logistics  processes  by  September  1999,  and  may 
not  be  able  to  meet  the  revised  goal  of  November  1999. 
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Additional  Air  Force  Measures  to  Mitigate  Risk.  Inspector  General,  DoD, 
Report  No.  00-021  addressed  measures  taken  by  the  Air  Force  to  mitigate  risk 
of  Y2K-related  system  failures.  The  Air  Force  policy  requires  that  100  percent 
of  the  code  that  impacts  mission-critical  processes  be  scanned  using  two  code 
scanning  tools.  The  code  scanning  effort  initiated  by  the  Air  Force  should  assist 
in  uncovering  remaining  Y2K-related  errors,  as  well  as  providing  system 
managers  the  opportunity  to  validate  and  fix  those  errors,  and  retest  systems  as 
needed.  We  were  advised  by  Air  Force  officials  that  scanning  of  code  that 
impacts  mission-critical  automated  information  systems  is  an  ongoing  effort. 


Conclusion 

The  Air  Force  did  not  comply  with  the  DoD  Management  Plan  and  the  Logistics 
Capstone  Plan  in  its  efforts  to  plan  and  manage  its  logistics  Level  I  end-to-end 
testing.  Although  the  Air  Force  identified  22  core  processes  and  planned  to  test 
14  of  the  core  processes  and  22  mission-critical  systems  during  Level  I  end-to- 
end  testing,  it  did  not  develop  sufficiently  detailed  plans  for  conducting  the  tests. 
Without  such  detailed  Level  I  end-to-end  test  plans,  there  was  no  standardized 
testing  approach.  Also,  without  the  detailed  test  plans,  analysis  to  ensure  that 
test  goals  were  met  and  that  testing  was  successfully  accomplished  for  all  critical 
core  processes  and  related  systems  was  hampered.  In  order  to  mitigate  the  risk 
of  Y2K-related  system  failures,  the  Air  Force  needs  to  test  contingency  plans 
for  the  mission-critical  systems  that  were  to  be  included  in  Level  I  end-to-end 
testing  and  put  a  risk  management  plan  in  place  for  each  of  the  Air  Force  core 
processes. 


Recommendations 

We  recommend  that  the  Chief  Information  Officer,  Department  of  the  Air 
Force: 


1.  Ensure  that  contingency  plans  for  the  22  mission-critical  logistics 
systems  that  were  to  be  included  in  Level  I  end-to-end  testing  are  tested. 

2.  Ensure  that  a  risk  management  plan  that  includes  a  risk  assessment 
and  mitigation  plan  for  each  of  the  Air  Force  core  logistics  processes  for  the 
Level  I  end-to-end  testing  is  developed  and  provided  as  soon  as  possible  to  the 
Deputy  Under  Secretary  of  Defense  (Logistics  and  Materiel  Readiness).  The 
risk  management  plan  should  be  based  on  probability  of  occurrence  and 
consequences  of  occurrence,  and  list  the  mitigation  for  a  particular  risk. 

Management  Comments  Required 


The  Air  Force  did  not  comment  on  a  draft  of  this  report.  We  request  that  the 
Chief  Information  officer,  Department  of  the  Air  Force,  provide  written 
comments  on  this  final  report. 
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Appendix  A.  Audit  Process 


This  is  one  in  a  series  of  reports  being  issued  by  the  Inspector  General,  DoD,  in 
accordance  with  an  informal  partnership  with  the  Chief  Information  Officer, 
DoD,  to  monitor  DoD  efforts  to  address  the  Y2K  computing  challenge.  For  a 
listing  of  audit  projects  addressing  the  issue,  see  the  Y2K  web  pages  on  IGnet  at 
http :  / /www .  ignet .  gov . 


Scope  and  Methodology 

Work  Performed.  We  reviewed  the  Y2K  test  planning  efforts  of  the  Air  Force 
for  the  logistics  functional  end-to-end  testing.  We  evaluated  the  Y2K  planning 
efforts  and  compared  those  efforts  with  the  criteria  contained  in  the  DoD 
Management  Plan  and  the  Logistics  Capstone  Plan.  We  reviewed  Public  Law 
105-261,  Section  334;  the  Deputy  Secretary  of  Defense  memorandum  of 
August  24,  1998;  the  DoD  Management  Plan;  the  Logistics  Capstone  Plan;  the 
Air  Force  Level  II  Test  Plan;  the  draft  Air  Force  Assessment  Plan;  and  other 
guidance  regarding  the  testing  of  critical  core  logistics  processes  and  mission- 
critical  systems.  Documents  reviewed  were  dated  from  October  1998  through 
September  1999.  We  interviewed  personnel  within  the  offices  of  the 
DUSD(L&MR),  the  Air  Force  Deputy  Chief  of  Staff  for  Installations  and 
Logistics,  and  the  Air  Force  Y2K  Program  Office.  We  also  interviewed 
contractor  representatives  involved  with  end-to-end  testing. 

Limitations  to  Scope.  Our  review  was  limited  to  test  planning  accomplished  by 
the  Air  Force  for  Level  I  logistics  end-to-end  testing. 

DoD-Wide  Corporate-Level  Goals.  In  response  to  the  Government 
Performance  and  Results  Act,  DoD  established  2  DoD-wide  corporate-level 
goals  and  7  subordinate  performance  goals.  This  report  pertains  to  achievement 
of  the  following  goal  and  subordinate  performance  goal. 

Goal  2:  Prepare  now  for  an  uncertain  future  by  pursuing  a  focused 
modernization  effort  that  maintains  U.S.  qualitative  superiority  in  key 
warfighting  capabilities.  Transform  the  force  by  exploiting  the 
Revolution  in  Military  Affairs  and  reengineering  the  Department  to 
achieve  a  21st  century  infrastructure.  Performance  Goal  2.2: 

Transform  U.S.  military  forces  for  the  future.  (00-DoD-2.2) 

DoD  Functional  Area  Reform  Goals.  Most  major  DoD  functional  areas  have 
also  established  performance  improvement  reform  objectives  and  goals.  This 
report  pertains  to  achievement  of  the  following  objectives  and  goals  in  the 
Information  Technology  Management  Functional  Area. 

•  Objective:  Become  a  mission  partner.  Goal:  Serve  mission 
information  users  as  customers.  (ITM-1.2) 

•  Objective:  Provide  services  that  satisfy  customer  information  needs. 
Goal:  Modernize  and  integrate  Defense  information  infrastructure. 
(ITM-2.2) 
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•  Objective:  Provide  services  that  satisfy  customer  information  needs. 
Goal:  Upgrade  technology  base.  (ITM-2.3) 

High-Risk  Area.  In  its  identification  of  risk  areas,  the  General  Accounting 
Office  has  specifically  designated  risk  in  resolution  of  the  Y2K  problem  as  high. 
This  report  provides  coverage  of  that  problem  and  of  the  overall  Information 
Management  and  Technology  high-risk  area. 

Audit  Type,  Dates,  and  Standards.  We  performed  this  program  audit  from 
June  through  October  1999  in  accordance  with  auditing  standards  issued  by  the 
Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD.  We  did  not  use  computer-processed  data  for  this  audit. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD.  Further  details  are  available  on  request. 

Management  Control  Program.  We  did  not  review  the  management  control 
program  related  to  the  overall  audit  objective  because  DoD  recognized  the  Y2K 
issue  as  a  material  management  control  weakness  area  in  the  FY  1998  Annual 
Statement  of  Assurance. 
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Appendix  B.  Summary  of  Prior  Coverage 


The  General  Accounting  Office  and  the  Inspector  General,  DoD,  have 
conducted  multiple  reviews  related  to  Y2K  issues.  General  Accounting  Office 
reports  can  be  accessed  over  the  Internet  at  http://www.gao.gov.  Inspector 
General,  DoD,  reports  can  be  accessed  over  the  Internet  at 
http://www.dodig.osd.mil.  The  reports  most  relevant  to  the  subject  matter  of 
this  report  are  listed  below. 


General  Accounting  Office 

General  Accounting  Office  Report  No.  GAO/AIMD-99-172  (OSD  Case 
No.  1823),  “Defense  Computers:  Management  Controls  Are  Critical  to 
Effective  Year  2000  Testing,”  June  30,  1999. 


Inspector  General,  DoD 

Inspector  General,  DoD,  Report  No.  D-2000-040,  “Navy  Logistics  Year  2000 
End-to-End  Test  Planning,”  November  16,  1999. 

Inspector  General,  DoD,  Report  No.  D-2000-036,  “Defense  Logistics  Agency 
Logistics  Year  2000  End-to-End  Test  Planning,"  November  12,  1999. 

Inspector  General,  DoD,  Report  No.  D-2000-033,  “Army  Logistics  Year  2000 
End-to-End  Test  Planning,”  November  5,  1999. 

Inspector  General,  DoD,  Report  No.  00-021,  “Air  Force  Logistics  Year  2000 
End-to-End  Test  Planning,”  October  26,  1999. 

Inspector  General,  DoD,  Report  No.  00-002,  “Year  2000  End-to-End  Testing: 
Logistics  Capstone  Plan,”  October  1,  1999. 
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Appendix  C.  Air  Force  Core  Logistics  Processes 
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Appendix  D.  Air  Force  Mission-Critical  Logistics 
Systems 
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Appendix  E.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Deputy  Under  Secretary  of  Defense  (Logistics  and  Materiel  Readiness) 

Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Deputy  Chief  Information  Officer  and  Deputy  Assistant  Secretary  of  Defense  (Chief 
Information  Officer  Policy  and  Implementation) 

Principal  Director  for  Year  2000 


Joint  Staff 

Director,  Joint  Staff 


Department  of  the  Army 

Chief  Information  Officer,  Army 
Auditor  General,  Department  of  the  Army 
Inspector  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 

Chief  Information  Officer,  Navy 

Auditor  General,  Department  of  the  Navy 

Inspector  General,  Department  of  the  Navy 

Inspector  General,  Marine  Corps 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Chief  Information  Officer,  Air  Force 
Auditor  General,  Department  of  the  Air  Force 
Inspector  General,  Department  of  the  Air  Force 
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Unified  Commands 


Commander  in  Chief,  U.S.  European  Command 
Commander  in  Chief,  U.S.  Pacific  Command 
Commander  in  Chief,  U.S.  Central  Command 
Commander  in  Chief,  U.S.  Transportation  Command 
Commander,  Military  Traffic  Management  Command 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 

Chief  Information  Officer,  Defense  Contract  Audit  Agency 
Director,  Defense  Information  Systems  Agency 

Inspector  General,  Defense  Information  Systems  Agency 
Chief  Information  Officer,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 

Chief  Information  Officer,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 
Inspector  General,  National  Imagery  and  Mapping  Agency 
Inspector  General,  National  Reconnaissance  Office 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Office  of  Information  and  Regulatory  Affairs 
National  Security  Division  Special  Projects  Branch 
Federal  Chief  Information  Officers  Council 
General  Accounting  Office 

National  Security  and  International  Affairs  Division 
Technical  Information  Center 
Accounting  and  Information  Management  Division 

Director,  Defense  Information  and  Financial  Management  Systems 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Governmental  Affairs 

Senate  Special  Committee  on  the  Year  2000  Technology  Problem 

House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 


15 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member  (cont’d) 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform 

House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Committee  on  Science 
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Audit  Team  Members 


The  Readiness  and  Logistics  Support  Directorate,  Office  of  the  Assistant 
Inspector  General  for  Auditing,  DoD,  prepared  this  report.  Personnel  of  the 
Office  of  the  Inspector  General,  DoD,  who  contributed  to  the  report  are  listed 
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Shelton  R.  Young 
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Kevin  T.  O’Connor 
William  E.  Shimp 


n 


INTERNET  DOCUMENT  INFORMATION  FORM 


A .  Report  Title:  Air  Force  Level  Logistics  Year  2000  End-To-End  Test 
Planning 

B.  DATE  Report  Downloaded  From  the  Internet:  02/07/99 


C.  Report’s  Point  of  Contact:  (Name,  Organization,  Address,  Office 
Symbol,  &  Ph  #):  OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions) 

Inspector  General,  Department  of  Defense 
400  Army  Navy  Drive  (Room  801) 

Arlington,  VA  22202-2884 


D.  Currently  Applicable  Classification  Level:  Unclassified 

E.  Distribution  Statement  A:  Approved  for  Public  Release 

F.  The  foregoing  information  was  compiled  and  provided  by: 

DTIC-OCA,  Initials: _ VM _ Preparation  Date  02/07/99 


The  foregoing  information  should  exactly  correspond  to  the  Title,  Report  Number,  and  the  Date 
the  accompanying  report  document.  If  there  are  mismatches,  or  other  questions,  contact  the 
above  OCA  Representative  for  resolution. 


